Real-time fish animation generation by monocular camera

Fish animation generation is an interesting topic since it plays an important role in building virtual underwater worlds. Accurate motion capture and flexible retargeting of fish is difficult, in particular with the challenges of underwater marker attachment and feature description for soft bodies. Little research into this problem has been published and real-time fish motion retargeting with a desirable motion pattern remains elusive. Motivated by our goal of achieving high-quality data-driven fish animation with a light-weight, mobile device, this paper develops a novel framework of motion capturing, retargeting, and fine tuning for a fish. We demonstrate a markerless technique for the motion capture of an actual fish using a monocular camera. The elliptical Fourier coefficients are then integrated into the contour-based feature extraction process to analyze fish swimming patterns. This novel approach can obtain motion information in a robust way, utilizing the smooth medial axis as the descriptor for a soft fish body. For motion retargeting, we propose a two-level scheme to transfer the captured motion into new models, such as 2D meshes (with texture) generated from pictures or 3D models designed by artists, regardless of the body geometry and fin proportions amongst various species of fish. Both the motion capture and retargeting processes operate in real time. Hence, the system can obtain video sequences of real fish using a monocular camera and simultaneously create fish animation with variation. In addition, a motion fine tuning method is provided for animators to efficiently refine the retargeted frames in an interactive manner. It can enhance the final output animation to an appropriate level of fidelity. © 2017 Elsevier Ltd. All rights reserved.